IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 



In re Application of: : Confirmation No.: 2283 

Cosattoetal. : Attorney Ref.: 2000-0042-CON 

Serial No.: 10/662,550 : Art Unit: 2628 

Filed: September 15, 2003 : Examiner: Daniel F. Hajnik 

FOR: AUDIO- VISUAL SELECTION PROCESS FOR THE SYNTHESIS OF PHOTO- 
REALISTIC TALKING-HEAD ANIMATIONS 

AFFIDAVIT UNDER 37 CFR 1.131 

Honorable Commissioner of Patents & Trademarks 
Alexandria, VA 22314 

Dear Sir: 

I, Wendy Koba, being duly sworn, depose and state: 

1. I prepared, drafted, and filed patent application 2000-0042, now issued as U.S. 
patent 6,654,018, filed 29 March 2001. Application 2000-0042 is the parent of 
the above-identified patent application. 

2. Exhibit #1 is a handwritten listing of projects I billed out between 12 June 2000 
and 29 March 2001. 

3. Exhibit #2 is a set of email correspondence between AT&T, Eric Cosatto, and me 
between April 2000 and March 2001. 



Application/Control Number: 10/662,550 Docket No.: 2000-0042-CON 

Art Unit: 2628 

4. On 5 April 2000, 1 received a letter from Ann E. Taylor of the AT&T Intellectual 
Property group with instructions to prepare and file the application 2000-0042 
within 6 months, or roughly by October 5, 2000. See Exhibit #2, page 2. 

5. On 1 1 April 2000, 1 sent a letter to inventor Eric Cosatto acknowledging that the 
patent submission had been assigned to me. See Exhibit #2, page 3. 

6. On 17 July 2000, 1 sent a letter to Eric Cosatto acknowledging a publication date 
of Eric Cosatto's paper on August 1, 2000. See Exhibit #2, page 4. 

7. On 24 August 2000, Eric Cosatto wrote me an email requesting the status of the 
application. See Exhibit #2, page 4. 

8. On 2 October 2000, 1 wrote an email to Eric Cosatto explaining that two 
"emergency" cases had taken priority over Eric Cosatto's application. See 
Exhibit #2, page 5. 

9. On 8 November 2000, Eric Cosatto wrote me an email requesting a status update 
of the application; see Exhibit #2, page 5. 

10. In mid-November 2000, my son was diagnosed with Type I diabetes. I spent 
several weeks attending to this matter. 

11. In December 2000, AT&T "paced" filings in Dec. to hit a specific number. 

12. On 2 March 2001, 1 wrote an email to Eric Cosatto with the application draft. See 
Exhibit #2, page 6. 

13. Later on 2 March 2001, Eric Cosatto wrote me a letter with a revised version of 
the application draft. See Exhibit #2, page 7. 
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14. On 14 March 2001 , 1 sent Eric Cosatto an email including the application, 
drawings, and signature papers for filing the patent application. See Exhibit #2, 
page 8. 

15. On 19 March 2001, Eric Cosatto wrote a letter to the other inventors with the final 
draft of the application. See Exhibit #2, page 9. 

1 6. On 26 March 2001, inventor Gerasimos Potamianos wrote an email to Eric 
Cosatto indicating that assignment papers were sent. See Exhibit #2, page 10. 

1 7. On 29 March 2001 , 1 wrote an email to Ann E. Taylor indicating that application 
was filed. See Exhibit #2, page 11. The application was filed 29 March 2001 and 
assigned Appl. No. 09/820,396. 

All the statements made herein are true and are made on information believed to be true; and 
further these statements are made with the knowledge that willful false statements and the like so 
made are punishable by fine or imprisonment, or both, under 18 U.S.C. 1001 and that such 
willful false statements may jeopardize the validity of the application or any patent issued 
thereon. 





Date 
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MessagcRead Page I of 2 



Folder: Inbox 

_<J_>J Delete | Forward [ Reply j Reply All | Mov'eTd: | [AT&T 



C. 



Date: Thu. 24 Aug 00 14:02PM MDT 

L - n3 SficTSii55^ Add To Address Book 



To: <wendykoba@usa.net> 
Subject: IOS No. 2000-0042 
More Details I Print Preview 



Hi Wendy, 

Just checking for news regarding this patent application. 
Let me know where it stands. Thank you, 

Eric. 



II 

| Eric Cosatto < eric@research.att.com > | 
I AT&T Labs - Research | 

| Image Processing Software & Technology Research Dept | 
I Room 3-124, Red Bank, NJ 07701-7033, USA | 
| Tel: ++1 (732) 345 3337 ; Fax: ++1 (732) 345 3032 | 
j WWW: http://wwwTesearch.att.com/info/eiic | 

I ■ I 

— Original Message — 
From: <wendykoba@usa.net> 

To: <eric@research.att.com> . . — 

Sent Monday, July 17, 2000 7:55 AM /-r . \ v \ * # tp/vOO 

Subject: Re: [Re: photo realistic ) y' 




The answer is "yes" - thanks for letting me know. 
| Usually, if it's a "rush" that info's included on what Tom 
| sends me - 1 sure he did and I just didn't realize it. 
| I'll read it over this AM and see if I can get a 
| "rea" patent together by the end of the week for you to 
review. 

| We can always file a provisional before the publication 
j date, but it's not my preferred way to work. 
| Wendy 

j "Eric Cosatto" <eric@research.attcom> wrote: 

| Hi Wendy, 

I 

1 1 got word from Tom Restaino that the patent "Audio-Visual 
Unit Selection 

| For Concatenate Synthesis Of Talking Head Animations" 
(IDS No. 2000-0042) 
| was 

j sent to you for filing. 

j The publication date is August 1st. Do we need to have 
http^/www.netaddress.com/tpI/Message/5 1 4 AURFED/Read 8/24/00 
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Message.Read 



Page 1 of 3 



Folder: Inbox 



JlJ-L] De^te Forward Reply Reply All Move To: [ [AT&T 



/ Date: Wed, 08 Nov 00 14:11PM MST 
V From: 

\. "Eric Cosatto" <eric@research.att.com> 

To:^wewlykcibo@usa,not^ — — 
Subject: Re: (Re: (Re: photo realistic ]] 
More Details J Print Preview [ 

Hi Wendy. 

Can you update me on the progress of this patent Thanks. 
Eric Cosatto (PLEASE NOTE THE NEW ADRRESS) 



Add To Address Book 



II 

| Eric Cosatto < e.-ic@research.att.com > | 

| AT&T Labs - Research | ' ■< . ,j$> 

j Image Processing Software & Technology Research Dept. } N /-y (y 

I Room A5-4F32, Middletown, NJ 07750, USA | 1 ^ Cr 

| Tel: ++1 (732) 420 9142 ; Fax: ++1 (732) 368 9475 j r 

| WWW: http://www.research.au. conVinfo/eric | ^ 

I I 



— Original Message — 

From: <wendykoba^usa.net> 

To: <eric@research.an.com> 

Sent: Monday, October 02, 2000 7:24 AM 

Subject: Re: (Re: (Re: photo realistic ]] 



X 



I Eric, 



HI - this email did get thru. For some 
unknown "net" reason, I had not been receiving 
about half my mail during August. I have two 
"emergency" cases to file for AT&T by Wed., then 
I will be getting back to your application. I know 
you presented your paper (and I think it was the 
same information I received for your application). If 
your paper was a newer/enhanced, etc. version, I'd 
appreciate a copy (email is fine). 

Wendy 

"Eric Cosatto" <eric@research.att.com> wrote: 
Hi Wendy, 

I havn't heard back from this. I sent you an email on 8/24 
but I did not get any answer. Let me know if you receive 
this email. Thanks. 



hup://mw.net^dress.^ 11/9/00 
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Message.Read 
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j Eric Cosatto < eric@research.att.com > | 
j AT&T Labs-Research, Room A5-4F32 | 

> j Middtetown, NJ 07748, USA 

I < 

| Tel: ++1 (732) 420 9142 ; Fax: ++1 (732) 368 9475 | 
I WWW: http://www. research.att.com/info/eric | 



— Original Message — , 

/ From: <wendykoba@usa.net> \ Yll L M / /VV 

[ To: <eric@research.att.com> \ Ml f^rOfl Cx r OLL^I 

\ Sent: Friday, March 02, 2001 9:32 AM j \ ' 



\ Subject: Photo-realistic patent draft 



Eric, - . ' 

attached please find a draft, including claims, 
for your application. 

please review and forward comments to me in the 
easiest manner for you - email is fine, and so is 
fax (fax #61 0-346-8189). 

thanks 

Wendy 



Get free email and a permanent address at http://www.netaddress.com/?N=1 
D cosatto 0042_c.doc (70K) 



< | > | Delete | Forward | Reply j Reply AH [ Move To: [ [AT&T H 



© 1998-2001 USA.NET®. tnc All Rigms Reserved, U.S. Patent # 5.937.161. 
Users of this sits agree to be bound by USA.NET Net@ddress* Subscriber Agreement, 
USA.NET adheres to a strict Privacy Policy to protect our customers. 



hitp://k2.neiaddress.corr^ 3/5/01 
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Folder: Inbox Message 6 of 32 (OLD) 



>X BelfeTTf^oTwa^ I Move To: [[AT&T 



Oate: Fa 02 Mar 01 15;04PM MST 

Ff ° m: "Eric Cosatto" <eric@research:a»:com> Add To Address Book 



i:-<wendytoba@asd^et> ouAo-^Auoi 

Cc: "Hans Peter Graf <hpg@research.atLcom> arJLr t*u^u>^ '* ' ~ ' 
Subject: Re: Photo-realistic patent draft ^ ~*^ r 

More Details | Print Preview | 



Hi Wendy, ^^rro^r^ 

Please find attached a new version of the draft (no claims). Use "Track Changes" / . */$ozJL ^ o-fy^ 

in Word to see the / *' : — 
modifications. 




Regarding the claims, I'd (ike to see the emphasis put on the "audio-visual 
unit-selection". This Q - V 

patent is about C OMBIN ING a ud io features (namely the phonemes) and visual 
features (PCA+geometric " " 

features) in the unit-selection (viterbi search). So. claim 1 should reflect 
that. 

: v The next major claim is the visual feature s. The innovation here is to COMBINE ,w~r^ ' . 5v\^> ^ £ 

high-level features " ~~ w A - ^""^ 

> ' /i)'""V such as geometric sizes and position of elements (e.g. widlh, h^iojTlofJhe ; ' w " v 

^ lips, or pose of the " — ■ / 

head) with pixel-based, low-4evel features such as (but not limited to) PCA. . / 

wavelets. " ' ~ 



Then comes the idea to NORMALIZE facial parts images in order to use these ^ ^ 

tow-level features, which • 

are otherwise very sensitive to image translation/rotation. The particular way 

of doing the ; b o >^p; 

normalization is prior art and should not be covered here. o tV* 3 -* < 

Again, This technique is applicable not only to mouth images but to any facial ' Oi-^ZL & * 

part Other facial "j,^" 
parts may be labeled with other attributes than the acoustic information, such p* 8 " . . 

as the emotional . °^ . 

expression. For example, for synthesizing animations of the eye area we can F * v 9^ 

record eyes while a 

person expresses different emotions; This information, together with the metric ^ 
describing the k **** 

visual appearance is then used to produce an animated sequence. The claims J? /v\s-U 
should cover this as / 
well. / 

Call me if you need more info. I'll be waiting for a new version of the claims? 
Thanks a lot — 

EricCosatto /TtoW**. 
http^/k2.nctaddress.com/tplAlessagc/214PXUJDU/Rcad 3/5/01 
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Wendy W. Koba 
Patents, Trademarks, & Copyrights 

P.O. Box 556 
Springtown, PA 18081 



Courier Address: 
Wendy W. Koba, Esq. 
540 Summit Lane 



Telephone: 610-346-7112 
Facsimile: 610-346-8189 
Email: wendykoba@usa.net 



Riegelsville, PA 18077 
Wednesday, March 1 4, 200 1 

Eric Cosatto 
AT&T Corp. 
Room A5-4F32 
200 Laurel Avenue 
Middletown, New Jersey 07750 

Re: Cosatto 2000-0042 "Audio-Visual Selection Process for the Synthesis of Photo- 
Realistic Talking-Head Animation*! 



Enclosed are the formal papers (the "Assignment" and "Declaration") for you to 
"sign-off* on. You will need to get in touch with Hans Graf to make an appointment to 
sign the paperwork. I have enclosed a priority mailer for you to send the signed 
paperwork back to me. I have mailed copies of the formal paperwork directly to 
Gerasimos Potamianos and also Juergen Schroeter. 

If you are satisfied with the application and drawings, please contact someone in 
the IP-Law group to make an appointment to "sign off" on the paperwork. 

Please do not sign anything before meeting with someone there, since the 
Assignment must be notarized (the secretaries in the patent group are notaries). 

If you have any questions, please do not hesitate to give me a call or send me an 
e-mail message. 



Dear Eric: 



Sincerely, 



Wendy W. Koba 



Enclosed: 



Assignment and Agreement 
Declaration and Power of Attorney 
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Message.Rcad 



Page 1 of 



Folder: Inbox 



Message 3 of 68 (NEW) 



< | > | Delete j Forward Reply | Reply All | Move To: | [AT&T 



Date: Mon. 19 Mar 01 17:02PM MST 

Ff ° m: "Eric Cosatto- <eric@research.att.com> Add To Address Book | 

To: "Wendy Koba** <wendykoba@usa.net> 
Cc: "Hans Peter Graf* <hpg@research.attcom>, "Juergen Schroeter 
<jsh@research.attcom>, "Makis Potamianos" <gpotam@us.ibmxom> 

Subject AUDIO-VISUAL SELECTION PROCESS FOR THE SYNTHESIS OF PHOTO-REALISTIC 
TALKING-HEAD ANIMATIONS 

More Details I Print Preview j 



Hi All, 



Here's the final draft for the patent. You should be receiving the assignments 
papers from Wendy Koba Many thanks for your contributions. 

Eric Cosatto. 

D cosatto 0042Jinal.doc (89K) 



_lJ_lJ Pe^te | Forward Reply Reply All 



Move To: AT&T 



11 
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Date: Mon, 26 Mar 01 17:16PM MST 
From: 

"Gerasimos Potamianos" <gpotam@us.ibm.com> 
To: "Eric Cosatto" <eric@research.att.com> 



Add To Address Book j 



Cc: wendykoba@usa.net, jsh@research.att.com, hpg@research.att.com 
Subject: Re: AUDIO-VISUAL SELECTION PROCESS FOR THE SYNTHESIS OF PHOTO- 
REALISTIC TALKING-HEAD ANIMATIONS 

More Details I Print Preview I 



Hi Eric, 

the patent document looks really good. I already sent out the assignment 
papers to the lawyer. 

Best regards, 
-Makis 

"Eric Cosatto" <eric@research.att.com> on 03/19/2001 07:01:00 PM 

To: "Wendy Koba" < wendykoba@usa.net> 
cc: "Hans Peter Graf <hpg@research.att.com>, "Juergen Schroeter" 
<jsh@research.attcom>, Gerasimos PotamianosAWatson/IBM@IBMUS 
Subject: AUDIO-VISUAL SELECTION PROCESS FOR THE SYNTHESIS OF 
PHOTO-REALISTIC TALKING-HEAD ANIMATIONS 



Hi All. 

Here's the final draft for the patent. You should be receiving the 
assignments 

papers from Wendy Koba. Many thanks for your contributions. 
Eric Cosatto. 
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Application/Control Number: 10/662,550 
Art Unit: 2628 



Docket No.: 2000-0042-CON 



WENDY W. KOBA, ESQ. 
Patents, Trademarks & Copyrights 
P.O. BOX 556 
SPRINGTOWN, PA 18081 

Telephone: 610-346-71 12 
Facsimile: 610-346-8189 
E-mail: wendykoba@usa.net 

Thursday, March 29, 2001 
Ann E. Taylor 

Outside Counsel Coordinator 
AT&T Corp. 
P.O. Box 41 10 
Middletown, NJ 07748 

Re: Cosatto 2000-0042 "Audio-Visual Selection Process for the Synthesis of 
Photo-Realistic Talking-Head Animations'* 

My reference: A TT-0186 

Dear Ann : 

In accordance with your Outside Counsel Guidelines, enclosed please find a copy 
of the above-identified direct-managed application. This case was filed in the USPTO on 
March 29, 2001. 

Sincerely, 



Wendy W. Koba 



Enclosures: Specification & Claims 
Drawings (2 sheets) 

Declaration & Associate Power of Attorney 
Assignment & Agreement 
Assignment Recordation Cover Sheet 
Request for Non-Publication 
Disk (2000.0042doc) 
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Appl ication/Control Number: 10/662,550 
Art Unit: 2628 



Docket No.: 2000-0042-CON 



Wew Disclosure Information Fo .1 



Client / Division 



Managing Attorney 

Working Attorney 

(If Working Attorney Held is blank, use Managing Attorney) 



0l>irc 

tf opening < 

OC Firm / OC Attorney kbb^ 

^when appro 



□ AT&T Case □ AT&T Managed Case* [EfDirect-Managed Case* 

*OC case - If undecided at the time of opening disclosure, leave blank 



{when approved for filing) 
(Note: Unless the disclosure is already in the hands of an outside counsel firm, do NOT enter O/C 
Firm into the disclosure record until the disclosure is approved for filing by EPRT and sent to OC). 

IPRTeam: M W[ 

IPRT Technologies: (Primary Technology must match Team) 

Primary 3 

Secondary Q- 

Tertiary_ 



Ready for IPRTeam Review Q^C(Insert TTR date in Actions tab) □ No -Date Ready 
(Insert TTR date in Actions tab) 

Owner/Assignee: H^T&T Corp. 

□ AT&T Wireless Services, Inc. - AWS or STG No._ 

(Note: Call Ruby Tarr (425-702-3948) to obtain an AWS or STG number for 
all AT&T Wireless Services cases - Insert A WS/STG Number in 
TechJOther Us tab) 

□at&t/_ 



(Company or University sharing rights with AT&T) 



Please provide the following-information or supply it as an attachment to this form: 



Deadline Date (if any) 



Optional Information to be provided: r V /? x # / j / * 

Keywords: (Submit BEEF if not already in database) C»«u\>xaMo" , <hMc^ 

Government Contract Number, if any: ^pv^Lu^i , UW-fi^jJ^ v\\io « f TT^ j 

Legal Secretary - complete: ^tj^rhf 

□ Copy of Disclosure sent to EPRT Coordinator Via QEmail □ Fax □ Interoffice Mail ^ 

□ Copy of Wireless IPRT Disclosure sent to Rohini Garg Via OEmail □ Fax □ Interoffice 
^ E-Mail opening letter sent to inventor's E-level and copy to inventors 

Job Aid - PT1 Revised 1 1/07/1999 
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Application/Control Number: 1 0/662,550 
Art Unit: 2628 



Docket No.: 2000-0042-CON 




Ann E. Taylor Room E4-3B12 

Intellectual Property • Law 200 S. Laurel Avenue 

Middtetown, NJ 07748 U.SA 
PHONE (732) 420-5622 
FAX (732) 368-6932 
Email: 36taytor@lga.att.com 

April 5, 2000 



Wendy W. Koba, Esq. 
P.O. Box 556 
Springtown, PA 18081 

Dear Wendy: 

RE: Docket No. 2000-0042 

Title: "Audio-Visual Unit Selection For The Synthesis Of 

Photo-Realistic Talking Heads" 
Submitters: E. Cosano; H. P. Graf; G. Potamianos; J. Schroeter 



The above-identified submission has been approved for filing as a direct-managed case. 
Please prepare and file the formal application within the next six months and proceed with 
its subsequent prosecution as you deem best suited to protect AT&T's interests in this 
invention. The assignee is AT&T Corp. There will be no foreign filing of this application. 

Enclosed are copies of the disclosure material from our folder and a list of work and home 
addresses for each of the inventors. Please make your initial contact with the inventors in the 
next two weeks. Please con firm the proper inventorship of this invention. 

Relevant art determined by an electronic prior art search (less than one hour) should be 
identified for filing with the application. 

Should questions arise requiring AT&T attorney input, please call the AT&T contact 
attorney, Thomas Restaino, at 908-221-5720. For instance, please contact Tom with your 
recommendation if a Final Rejection is received, has been discussed with the inventors, and 
an Examiner Interview is unsuccessful in resolving any remaining problems. 



Please acknowledge receipt by signing and faxing this sheet to me at 732-368-6932. 

Received and acknowledged: \ \£Jlfs ^ iJ^-y-^Jt, Date: 2« i£i — 

Reference No.: CilSLj. i* 
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Application/Control Number: 10/662,550 
Art Unit: 2628 



Docket No.: 2000-0042-CON 



Wendy W. Koba 
Patents, Trademarks, & Copyrights 

P.O. Box 556 
Springtown, PA 18081 



Courier Address: 

Wendy W. Koba, Esq. 
540 Summit Lane 
Ricgelsville, PA 18077 



Telephone: 610-346-7112 
Facsimile: 610-346-8189 
Email: wendykoba@usa.nei 



Tuesday, April 11, 2000 
Eric Cosatto 

AT&T Corp. ? 
Room 3-124 t c i>~ S ^ 

1 00 Schultz Drive *>' \ J ' , ^ 

Red Bank, New Jersey 0770 1 iiP * 1 s ^ 

Re; IDS Wo. 2000-0042 

Title: "Audio-Visual Unit Selection For The Synthesis Of Photo-Realistic Talking 
Heads 19 



Dear Eric: 

The above- referenced patent submission has been sent to me to prepare. I will be 
contacting you soon if I need additional information to formulate a first draft. 

If you have any additional information on this submission beyond what you have sent to 
the IP-Law group, please let me know. 

Feel free to contact me at the above number or e-mail mc if you have any questions. 



Sincerely, 



Wendy W. Koba 
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